Papers with differential attention mechanism
DINT Transformer (2025.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show that the DINT Transformer improves accuracy and robustness across practical applications. |
| Approach: | They propose a differential attention mechanism that suppresses the impact of irrelevant contexts by computing DIF-Ference between two independent attention distributions. |
| Outcome: | The proposed architecture improves numerical stability and ability to capture global dependencies. |
Efficient OpAmp Adaptation for Zoom Attention to Golden Contexts (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models have shown significant promise in question-answering tasks . noisy reference documents hinder performance of LLMs, causing disproportionate attention to irrelevant content . |
| Approach: | They propose an adaptive large language model that allocates disproportionate attention to irrelevant documents . they use transformers to train the model and integrate it into pre-trained Transformer blocks . |
| Outcome: | The proposed model outperforms state-of-the-art models on noisy-context benchmarks. |